{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Poisson Regression\n",
    "\n",
    "Exponential family: $p(y;\\eta) = b(y)\\exp(\\eta^TT(y)-a(\\eta))$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (a)\n",
    "\n",
    "$$\\begin{align*} \n",
    "p(y;\\lambda) &= \\frac{e^{-\\lambda}\\lambda^y}{y!}\\\\\n",
    "&= \\frac 1{y!} \\exp(-\\lambda + y\\log \\lambda)\\\\\n",
    "&= \\frac 1{y!} \\exp((\\log \\lambda)^T y - \\exp(\\log \\lambda))\n",
    "\\end{align*}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we can take \n",
    "$$\\begin{align*}\n",
    "b(y)&=\\frac 1{y!}\\\\\n",
    "\\eta &= \\log \\lambda\\\\\n",
    "T(y) &= y\\\\\n",
    "a(\\eta) &= \\exp(\\eta)\n",
    "\\end{align*}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The canonical response function is \n",
    "$$ g(\\eta) = E[T(y);\\eta] = E[y;\\eta] = \\lambda = \\exp \\eta.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have \n",
    "$$ \\log p(y^{(i)} \\mid x^{(i)}; \\theta) = \\log b(y^{(i)}) + (x^{(i)})^T\\theta y^{(i)}- a((x^{(i)})^T\\theta)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Taking the differential with respect to $\\theta$ gives\n",
    "\n",
    "$$ \\mathrm d_\\theta\\log p(y^{(i)} \\mid x^{(i)}; \\theta) = y^{(i)}(x^{(i)})^T - \\exp((x^{(i)})^T\\theta)(x^{(i)})^T $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the gradient is given by $x^{(i)}(y^{(i)}-g(\\theta^Tx^{(i)})) =x^{(i)}(y^{(i)}-h_\\theta(x^{(i)})) $, which gives the update rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ \\theta \\mapsto  \\theta + \\alpha x^{(i)}(y^{(i)}-h_\\theta(x^{(i)}))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for stochastic gradient, where $\\alpha$ is the learning rate.\n",
    "\n",
    "Looking at a single coordinate $j$ this means that $\\theta_j$ updates as\n",
    "$$ \\theta_j \\mapsto \\theta_j + \\alpha x^{(i)}_j (y^{(i)}- h_\\theta(x^{(i)}))$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import src.util as util\n",
    "\n",
    "from src.linear_model import LinearModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_path='data/ds4_train.csv'\n",
    "eval_path='data/ds4_valid.csv'\n",
    "\n",
    "x_train, y_train = util.load_dataset(train_path, add_intercept=False)\n",
    "x_val, y_val = util.load_dataset(eval_path, add_intercept=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[1.        , 0.        , 0.99204749, 0.32574102],\n",
       "        [0.        , 1.        , 0.47046345, 0.37186578],\n",
       "        [0.        , 1.        , 0.49653283, 0.28985262],\n",
       "        ...,\n",
       "        [0.        , 1.        , 0.5731734 , 0.42840612],\n",
       "        [0.        , 1.        , 0.60116637, 0.94911474],\n",
       "        [0.        , 1.        , 0.32460144, 0.40509498]]),\n",
       " (2500, 4))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train, x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 2463097.,   645481.,   474123., ...,  1016002., 10620315.,\n",
       "          558245.]),\n",
       " (2500,))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train, y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a Poissonregression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PoissonRegression(LinearModel):\n",
    "    \"\"\"Poisson Regression.\n",
    "\n",
    "    Example usage:\n",
    "        > clf = PoissonRegression(step_size=lr)\n",
    "        > clf.fit(x_train, y_train)\n",
    "        > clf.predict(x_eval)\n",
    "    \"\"\"\n",
    "\n",
    "    def fit(self, x, y):\n",
    "        \"\"\"Run gradient ascent to maximize likelihood for Poisson regression.\n",
    "\n",
    "        Args:\n",
    "            x: Training example inputs. Shape (m, n).\n",
    "            y: Training example labels. Shape (m,).\n",
    "        \"\"\"\n",
    "        # *** START CODE HERE ***        \n",
    "        m,n = x.shape\n",
    "        \n",
    "        def gradient(theta):\n",
    "            return 1/m * x.T @ (y-np.exp(x@theta))\n",
    "        \n",
    "        theta = np.zeros(n)\n",
    "        lr = self.step_size\n",
    "        step = lr * gradient(theta)\n",
    "        \n",
    "        while np.linalg.norm(step,1) > self.eps:\n",
    "            theta += step\n",
    "            step = lr * gradient(theta)\n",
    "            \n",
    "        self.theta = theta + step\n",
    "\n",
    "        # *** END CODE HERE ***\n",
    "\n",
    "    def predict(self, x):\n",
    "        \"\"\"Make a prediction given inputs x.\n",
    "\n",
    "        Args:\n",
    "            x: Inputs of shape (m, n).\n",
    "\n",
    "        Returns:\n",
    "            Floating-point prediction for each input, shape (m,).\n",
    "        \"\"\"\n",
    "        # *** START CODE HERE ***\n",
    "        return np.exp(x@ self.theta)\n",
    "        # *** END CODE HERE ***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit a model to the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11.2993503 , 10.7993319 ,  2.00025994,  4.40057538])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = PoissonRegression(1e-7)\n",
    "clf.fit(x_train, y_train)\n",
    "clf.theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the predictions on the validation set. The red dots correspond to $(i,y^{(i)})$, and the blue crosses to the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1b4f8d1e9e8>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEDCAYAAAAlRP8qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2df5Ac5XnnPw+SWCEcicTiHGKzWpy44thJyWDBYZK4tIbkMFDRpcrekktFcnemtqxz7qJKDrBFgYNSkFrdKcklJNL5ZJdJrLKzOPGdKmc7J2WXxEkczOJDGIoYgwUKZyoosYKlE8ZGeu6P6R719HbP9Mx0T/+Y76dqamfnR8/78/s+7/M+79vm7gghhKg/55WdACGEEPkgQRdCiIYgQRdCiIYgQRdCiIYgQRdCiIYgQRdCiIZQqqCb2cfN7EUzezzDZ3/LzB4NHk+Z2T+PIo1CCFEXrMw4dDN7J3AK+AN3//E+vvcfgMvd/d8VljghhKgZpVro7v6XwLeir5nZD5vZF8zsETP7opm9OeGr7wM+NZJECiFETVhZdgIS+CjwAXf/upn9S+D3gXeFb5rZBuAyYKGk9AkhRCWplKCb2WuAa4AHzCx8eSL2sa3AZ9z9zCjTJoQQVadSgk7LBfTP7v62Lp/ZCnxwROkRQojaUKmwRXf/NnDUzN4LYC02hu+b2Y8C3w98qaQkCiFEZSk7bPFTtMT5R83seTN7P7ANeL+ZHQGeALZEvvI+4NOuIyKFEGIZpYYtCiGEyI9KuVyEEEIMTmmLouvXr/epqamyfl4IIWrJI4888o/ufnHSe6UJ+tTUFEtLS2X9vBBC1BIzey7tPblchBCiIUjQhRCiIUjQhRCiIUjQhRCiIUjQhRCiIUjQhRB9s3s3LN5xGKam4LzzYGqKxTsOs3t32SkbbyToQoi+ufKlw8zcu5HF5y4Ddxafu4yZezdy5UuHy07aWCNBF0L0zfSBW5hnhhnmuYu7mWGeeWaYPnBL2Ukba6p2fK4Qog4cO8Y0z7Gdvfw6d3Enu5jmQThmPb8qikMWuhCifyYnWWQze9nOnexiL9tZZDNMTpadsrFGFroQom8Wt+1n5t6NLTcLDzLNYsvtsu0I02UnboyRhS6E6JuH113H/M4jTG84CmZMbzjK/M4jPLzuurKTNtaUdh76pk2bXIdzCSFEf5jZI+6+Kek9WehCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQegq6ma02sy+b2REze8LM7k74zISZ/ZGZPW1mD5nZVBGJFUIIkU4WC/0V4F3uvhF4G3C9mV0d+8z7gRPu/iPAbwFz+SZTCCFEL3oKurc4Ffy7KnjEj2jcAtwfPP8McK2Z6dYlQggxQjL50M1shZk9CrwIHHL3h2IfeT3w9wDu/irwEvDahOvMmtmSmS0dP358uJQLIYToIJOgu/sZd38b8AbgKjP78dhHkqzxZQetu/tH3X2Tu2+6+OKL+0+tEEKIVPqKcnH3fwYeBK6PvfU8cCmAma0E1gHfyiF9QgghMpIlyuViM7soeH4BcB3wd7GPHQR+MXj+HmDBy7oVkhBCjClZbhJ9CXC/ma2gNQDMu/ufmtkuYMndDwIfA/7QzJ6mZZlvLSzFQgghEukp6O7+GHB5wut3RZ5/B3hvvkkTQgjRD9opKoQQDUGCLoQQDUGCLoQQDUGCLoQQDUGCLoQQDUGCLoQYmN27YfGOwzA1BeedB1NTLN5xmN27y07ZeCJBF0IMzJUvHWbm3o0sPncZuLP43GXM3LuRK186XHbSxhIJuhBiYKYP3MI8M8wwz13czQzzzDPD9IFbyk7aWJJlp6gQQiRz7BjTPMd29vLr3MWd7GKaB+GYTs8uA1noQojBmZxkkc3sZTt3sou9bGeRzTA5WXbKxhJZ6EKIgVnctp+Zeze23Cw8yDSLLbfLtiNMl524MUQWuhBiYB5edx3zO48wveEomDG94SjzO4/w8Lrryk7aWGJlnXK7adMmX1paKuW3hRCirpjZI+6+Kek9WehCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQegq6mV1qZotm9qSZPWFmv5zwmc1m9pKZPRo87iomudVGZ0MLIcoki4X+KvCr7v5jwNXAB83sLQmf+6K7vy147Mo1lTVBZ0MLIcqkp6C7+wvu/pXg+UngSeD1RSesjuhsaCFEmfTlQzezKeBy4KGEt99hZkfM7PNm9taU78+a2ZKZLR0/frzvxFaeY8eY5sH22dDb2RucDX2s7JQJIcaAzIJuZq8B/hjY4e7fjr39FWCDu28Efhf4H0nXcPePuvsmd9908cUXD5rm6qKzoYUQJZJJ0M1sFS0xP+DufxJ/392/7e6nguefA1aZ2fpcU1oDFrftb7tZdvGRtvtlcdv+spMmhBgDskS5GPAx4El3/82Uz/xg8DnM7Krguv+UZ0LrgM6GFkKUSc/z0M3sp4AvAl8FzgYv7wQmAdx9n5n9ErCdVkTMy8CvuPvfdLuuzkMXQoj+6XYees9b0Ln7XwFd7/jq7vcB9w2WPCGEEHmgnaJCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCCNEQJOhCiNLYvRsW7zgMU1Nw3nkwNcXiHYfZvbvslNWTWgq6GoEQzeDKlw4zc+9GFp+7DNxZfO4yZu7dyJUvHS47abWkloKuRiBEM5g+cAvzzDDDPHdxNzPMM88M0wduKTtptaSWgq5GIERDOHaMaR5kO3v5de5iO3uZ5kE4dqzslNWSnoJuZpea2aKZPWlmT5jZLyd8xszsd8zsaTN7zMyuKCa5AWoEQjSDyUkW2cxetnMnu9jLdhbZDJOTZaeslmSx0F8FftXdfwy4Gvigmb0l9pl3A28KHrPA3lxTGUeNQIhGsLhtf3uGvYuPtGfei9v2l520WtJT0N39BXf/SvD8JPAk8PrYx7YAf+At/ha4yMwuyT21AWoEQjSDh9ddx/zOI0xvOApmTG84yvzOI/zn/3OdAh8GoC8fuplNAZcDD8Xeej3w95H/n2e56GNms2a2ZGZLx48f7y+lEdIawcPrrhv4mkKI0XPbbTB9z3Xw7LNw9iw8+yzT91zHrZcr8GEQzN2zfdDsNcBfAPe4+5/E3vtfwG+4+18F//85cJu7P5J2vU2bNvnS0tLACRdCNJipqZaIM8929rKX7a3Ahw1HW+I/xpjZI+6+Kem9lRkvsAr4Y+BAXMwDngcujfz/BuCb/SZUCCGAIPDhuXbgw53sCgIfrOyUVZosUS4GfAx40t1/M+VjB4FfCKJdrgZecvcXckynEGKcUODDQGSx0H8SuBn4qpk9Gry2E5gEcPd9wOeAG4CngdPAv80/qUKIcWFx235m7t3YcrPwINMstgIhth1huuzEVZiegh74xbvOc7zliP9gXokSQow3rcCHw0wfOArHjOnJo8xvawU+SNDTybwomjdaFBVCiP7ptihay63/QohkdHDdeCNBF6JB6OC68UaCLkSD0MF1402mOHQhRE1Q/PZYIwtdiCah+O2xRha6EA1C8dvjjSx0IRqEDq4bbxSHLoQQNUJx6EIIMQZI0IUQoiFI0IUQoiFI0IUQoiFI0IUQogd1OSNHgi6EED2oyxk5EnQhhOhBXc7I0U5RIYToRU3OyJGFLoSoDGm+6htuKNmHXZMzciToQojKkOarvu6lz5Tqw17ctr/tZtnFR9rul8Vt+0fy+1mRoAshKkOar/pXHtpaqg+7NmfkuHspj7e//e0uhKgGc3PuCzsPuW/Y4G7mvmGDL+w85HNzI06ImTv4ndzt4H4nd7tD+7HsdbNqpX8EAEueoqs9LXQz+7iZvWhmj6e8v9nMXjKzR4PHXbmPOkKIQqlMWF6ar3rFiq4+7Mqkv2zSlD58AO8ErgAeT3l/M/Cnva4Tf8hCF6JCbNjgC2z29bzod3K3r+dFX2Bzy+IdIQs7D537bWinac81DyS+vrDzUKXSPwoYxkJ3978EvlXckNI86rKrTNSbXNvZsWNM82A7LG87e4OwvGM5p7o7ab7qw+ve092HXZH0l06a0nunFT5Fdwv9n4AjwOeBt3a5ziywBCxNTk6OaDwbPWlWRtuaECIHcm1ndbdw657+PqCLhZ6HoK8FXhM8vwH4epZrNtrlMkaNS5RIju2s7kZI3dPfD90EfeiwRXf/trufCp5/DlhlZuuHvW6t0fRPjIIc21ltwvJSqHv68yLTLejMbIrWwuePJ7z3g8A/uLub2VXAZ4AN3uPCjb4F3dRUa5Wdebazl71sb8XMbjgKzz5bdupEU1A7G0u63YKu51kuZvYpWn7y9Wb2PPARYBWAu+8D3gNsN7NXgZeBrb3EvOlU+c7ru3e3QrymD9zSsuQmJ1nctp+H113HbbeVnDjRF1VuZ6IcskS5vM/dL3H3Ve7+Bnf/mLvvC8Qcd7/P3d/q7hvd/Wp3/5vik11tqjz9U7xuc6hyOxPlkMnlUgSNdrlUGU3Thag13VwuOstl3NCCrRCFUuY+FAn6uFGTY0CFqCtlujV1g4sxQwtpQhRL68TIBLfmgaNwz7OF/rYs9DFDC2lCFEyJbk0tigohRJ4UHHgwVBy6EEKI7JTp1pTLRQghcqRMt6ZcLkIIUSMaEYeuM8aFEKI7tRF0bVkXQoju1EbQ0+4GPqq7fgsBmimK7JTRVmoj6NqyLqqAZooiK6W0lbQ7XxT96PuORboLkKgCaociKwW1FYq8Y9GoWNy2v+1m2cVH2u6XxW37y06aGCcSZooPc2XLCpMbRkQpwatQG0Fv8pb1PH1t8vEWTMLhZiv5Xsu4kBtGRCnjILw0073oR6NvEt0ned7gtoib5c7Nta7rGza4m7WmkjsP+dzcwJesLWnlu4cdcsOIDoq6cTVdXC4S9CqQp6+tAL/dON1RvReJgxubfY5b/U7udnC/k7tbXcus7OSKEinKEJKgVx0zd8hHEPK8VogWArsz4vLRjGm8kaBXhU9+sqMT+ic/2Xq94hZ6IYNEgxj1DEYzpvFmKEEHPg68CDye8r4BvwM8DTwGXNHrmj6Ggj639Su+MHF9q8iDx8LE9a3XK+5Dl4XenZFbzKqPsWZYQX8ncEUXQb8B+Hwg7FcDD/W6po+hoC+8bmuy0L5ua66CUIS4yCKsGJoxjTVDu1yAqS6C/t+A90X+/xpwSa9rjpugu1myVVWDTiifbcWQhT7WFC3ofwr8VOT/Pwc29bpmnQQ9F0HbsCHZqlInFH2iGdN4003Q89hYZAmvJR6ybmazZrZkZkvHjx/P4adHQy5nMtxzD4sT13duMpi4Hu65p7iEi1ypyqatOm6yq0rZNZ40pXfPbKE33+WSwxR3YcF9/fe97Auv29pyv7xua+v/heKSLfJFlvHgqOzyg4JdLjfSuSj65SzXrJWg57AINTfny8R7YcHlh64TJfmuG7GGURG/fxPKcihBBz4FvAB8D3geeD/wAeADwfsG/B7wDPBVMvjPvW6CXpHGKEqmpOiSRli3JUbmREU8elTDHLfWsiyHttCLeNRJ0BvRocTwlDWwV8SgGMq6LSEPYXoX1m5p/94Cm/0mDrpxxm/m/loaZxL0IWnCNE2cY9D6LG1gr0jc+TD5L6Psor+5wGZfxwm/gFO+jhN+M/fXNoZfgi5EhEHFpbSBvSIW+jDpKKXsYum9gFMO3rbM6+o+laCLsSCzaFRFIHuQ5jKYZV85Lr+KzBQyE0vvGk75zdzvxhnfw47auk+7CXptbnAhqkmV4osz7xeoyf1pw/x8+tvvZp4ZAGaYZ+vaz5cTd17GDRuGIUjvf+U/sob/x0q+xwW8zH/hV/kNdrLI5lrE8PdFmtIX/ZCF3gwqtWCc1fKuiYVetXRWqq4zEKZ3ln3t2c16XvSFtVtqvQaGXC6iMKokOhldArURpoq5OOoWHFC39GZFgi6Ko0qik3FwydLRKyEGVRosRWWQoItc6RC7wMJdxwm/lkOlik7lz5WvYRpE9egm6FoULYEqLSQOQnTxcZHN/DyfxYE7uId5ZphhnsVt+0eerjwPrZo+cEs7L3dxNzPMM88M0wduKSDlydTxEC5RMmlKX/RjnC302lteEVfAtRzytZxo56UpfspKuZKEiIBcLhWj7r7RcRC7uteRaCzdBF0ulzKoSRx0KiXEI3dzUxXhwlrctr/tZtnFR0p1JYlyqKNrVIJeBimCuHvdPbVoQGWIXbdNQ7ncgCSG/NeiiHZVOGmme9GPcXa5pPnQ91zzQC1861U4l6PDBSL3iCiCirYr5EOvFqmCeNG9lWxAlaCb334cfPpi9FS0XUnQ60JFG1AlkIUuRk1F25UEvS702YAqsZtxRHQL9ax9GKioJFVtV90EvRGLolVYjc4jDf0uNtZy0aZPwnJ9+Pcfbm3sWfFFFtnMwxf9bHuRUguYzaTsfj1Muyot7WlKX/QjTwu9CiNpHmno2+Ku6JQwT6pQt02myrO8Otd9kWmn8S6XKghbGWkYB597Feq2AqQJ77vfPbggz835ssiqPezwCzlVDdGsc90XmPbmC3oVhK2MNNS5wWelCnWbA8NawkWEuobX3MMOX8+LnXfzqUIbqnPdF5j2oQUduB74GvA08KGE9/8NcBx4NHjc0uuastCHp85T0sxUoW5zYOi6SiuHFSsGL5/INX+av2jfb7MyolmBuh94IK6qhQ6sAJ4B3gicDxwB3uLLBf2+XtfyggS9CsJWRhqq7P/MiyrUbS4M28HTLL7gMZAlGFzzZu53cP9p/qJSA2YV6n7QNFTWhw68A/izyP8fBj4c+0ypgl6UsPVz3XEQ1yLoVW6NKddhp+AFWeh72OHGGb+Z+zvcL1UYMCtR9wMOxEWmfVhBfw+wP/L/zXHxDgT9BeAx4DPApSnXmgWWgKXJycnhc1YwVbAQelGJRj8EdSjjXBjSQi/Kh76Gky2feeyadWk/w5Cp71TQjz+soL83QdB/N/aZ1wITwfMPAAu9rluLjUUV8OH1okxBzGUwqUEZ58Gw9VRUlMuojYGk35ydfspnp58auVGSqU4q2D4Ld7nEPr8CeKnXdUch6GHjCc9ICaencxfdm63BVHB0XkaJDS6XwaQOZTwAceGau+jeluV70b21nEnlRVKbWcsJXxe5ScrIjJIMfaeKM8hhBX0l8A3gssii6Ftjn7kk8vzngb/tdd1RCHo8LCv+t2elVGh0TrWmuG2ZIM5xq8+yzxfWbvFw0Wxh7RafnX4qXwHJo3wqVMZ5UiUhqJRbLqW+S2kDGYyJSpVdQB5hizcATwXRLncEr+0Cfi54/hvAE4HYLwJv7nXNkbhcIo0njLENF3+yNJgqdcrUtKzdsqwz7GFHh9XTYQXlmfYcrOsiyrgSi9kVGqjiZTzLvtZtA9duGb1IdYnWGfksrUJ11A/N31iURqzxhLG2WRtMpUbnNMtm7ZbkxbJA1Ndwyi/g1Lkp7RCNdVl5rFgR7Cw8OXCHKKKM0waJ2emnlv3WnmseaO2MzHvQrpIrKdZ2SnNxJKSlTAu9SgZbP4yvoA9poVeKFIGY47blgshmn+PW9mfzEpR4BwhD3uJREqV3iAEGv9zFpErWX0LbqdK6S1kDTKUMtj4YW0Ef2oeegWEbRebv9yMQwWfzttDjaWiHvK1YUa0O0WVan1iGBVjSvay/kYpJStvJmuc801qlKJe6MraC3ivKZZiQr5Bhp21Zv9/P7yzsPFSMD71KboRupA1+Kb7aIqzVXiI4yul+N6s4S577SWtdrd5RkFfZjK2g9yKXTjXs1Drj9/td6Judfir/KJeUtIYDZFU6cb8LyO18cGvH/2VYzUXsQIx/Z2HtFl/HCZ9lX7Z230day/ZLV3lAyatsGiXouVZYHn7OYa3WHKzeUTXitAaZdbfiqNKZ9juz008tS+caTvqeax7oaAt72OFz3FqsGA1Y73mIQt/10E9aS147KHtA6UpOZdMoQc+1wvJwIfRRSYkdae0Wn2XfUJWcpUzyENNhb26dV90Nmpee38tZjLr+3qC/VYZg9vObZbvlqrQYHSensmmUoOdaYTlcq1/fdiEr/BnyUajlkrWh5lDehd6UIWcx6lbmA9dHCYLZV1rLFtScyqeQ2aQs9ARybNAjn74mVGh7R+cwDSdLmRTZ0bJeO8eNSP3clKGQSKIsdLleFc/ZTqOftJbu8sipfIrIh3zoSeTYoEe+gFKUdZWlTAq0XLJszpmb82WLku2FuX7qLpLXrDdlKCKSKBMF1PcoIk6G6RdlL0oOUod5uEKz5FtRLgkUaQEU3hhzcjkMIqhFWi7RhcW0clvYmR5KmbSDM7XcA5Hs66YMGfOee/0XYE2Pwlpe2DnYsbpli/mgaUgtJ7LH6o9yZtIoQe9VYcMcM1p0peRx/WEEtZ/fTj0t0G5vf/dm7vcLOZl5MS88QyS+ezMp+qSbj7bvmzKUtFBXuvExxMLrIDe+KLr/jPrcnb7KboSusEYJei/SGlWm0LqCLblcGuSADaff346XY3ybf2ghZ74HZTdRjeWp2w7UgazHkhbqirRYM4nnoAPZILMg98LLedCZQ0/MfI5b2/kNj0boK1Z/hEZDIwW97xC62K26EkWDzjNQ0iplFNOr1F2uQXz0tRwqpOG0yzVSXqElHl+I7HUuTkcdBeW0jhN+LYc6vxfrDB3XjZVvVpGMfq7d8fuMLy9sg1YeZBHPISz0ftcp3L14URtw5jDIdW/kYH8nUspCH07Qu/m9EhtV8OglGpkOauriPsiro8ejOaJ/+9m2PejvhuUSt8Tj/3cTx+i1QjFfG/Ght7+X0BmGPTAr+tvRHaBhVNHsxCda1leXm5508/uPLGojjQziGa/LrMfmxttelkgidy9e1AadOfSgn/z2s2lNPvR+SGs8aTfNjbx+La2OGhWNdZzwmzh4rtNHRWftls6Gbxb4rk91TNGSKjCP8LSoRVz4yXQpvxs2+DWc7BwMV6xIz0/kWmGZh+nO4kb5Gb4wuKWX5hcNTlzMdGBbcI3cDznrQV7+8eh15rjN96y6ve1GmONW3859HWsg4W8MGuufNojGrz9weUT68E/waNeZQ79HZcQNwZu53+e4NfPsvK+F/SFppqCnWChxSz3Jhx61FsPR/gJOnRPyyDGrC2xuhy+1Ky1oWGs54RO87Kt4JbWjD+KeSWpgYQO+lkPnRLGIhhOzgsIOM9BRuRmn4EkCsp37HM52DB59nRmT9tuRtPd0HcWuMaoF1SxtJuvieHiSYXQgW8cJfxN/53DWt3Nf4m+kLoon3EIvyT0YnpOfxzEKcQu61TbO+Nv58rk2GTMs+u53WWcXJa3HRGmmoKcUbFqn74hyCSr4Qk76Kl5Zbnn1Yd1dEFjpF4Rhgznsjow34JaletZ/hi/k2oC6xd+u4aRfwVLbqp7jVt+z6nafnfhE67Z3vSyeYOALp/nRiJhE11TCzCBp8StzJ00r95jrretNT0qy0Pu1vruFr8ZdRus44St5xcHb7SlLu+wmkKmBCHmdMx+UxxpO+s/wBTfO+Hbua4v7Kr7T/r89G+gzjjzzAJBgKIQD1iisc/eGCvqg/kF37+0y6WHdhW6HqC87vkmm3eFiAnIth3pbeJFFmngD7rYI1O80M2lavZrTvprTvocdHTOZPexod5BeVlZ0QFrLCb+Qk+1rpPqgExZG22ZxlwXvtJ22s9NPJUdErLq9nY+ohZ60SL7nmgeWlUeY/j3XPJDL8cuJZCmLjDuRo67DO7nbz+d0h/85GtnRtV12G2S6DJ7LFu+h/zJKKY9O19zZ5bNHCjjzvct6T09XbU40UtDjEQgLbPbVnPar+euOKd6NG44sPzx/7Ra/ii+lL2p28/1G3BGreCU1hCppQTCzhRcLo4o24KTFu5B+pplpC0E3ctD3rLq9I+3hTCazdRqxqHoNfNHv9FxoThho01xss9NP+Z5rHugQ9XDAupCT7fWS6DQ+PitYw0m/ccORZeVxEwf9Qk4lzhiy7AnoSYbZSuJaTZofuMNtdM6NFeY5k1sr5WbkSbOe0GgJB8BVvOJrwtlZpJ9kHhDTZuOxPhKW14Wx3+oVCtsPaesEobEywcu+mtOdwp7zwmgjBd19uYCFftdwKhm/WXLSNDRp8SY6dQ3FeCKw0kLL+QqW2tZb1MJdxSutDh3xs6/gu76K73RUcuJAE7qGMmyRz7JdOWzE4WagObu94/9Z9vk6TvgUz7Sn4NGZSBhieEFkJpNm5cT9qHPc2l7UvJn725Za2gwlHlEyyz5fxXfOLdqBL9i7AjE9ucwCTVysXbvFb+JgYjja7MQnzn0uUjaJ90ftYwG+oy31IfLdwizjIXVJg2q3SI0kYQ2vuZ372sK8hpO+/Yc+u2zWdiGnOtpkPKggvN5qTvsFnPLVnO5wgexhh8NZN15t+9XXcNK3296OWO/wt+ILjGnnt8cHpGh7iw/0rTScWbZm0E8Mey8jMmroxUNz89yT0FhBT+poYaVGQ5riI/WNHPQFpjuE6yYOdjTsdgeM+NtX8N1lneRCTnaEEMY79BUsObhP8HKqa6Ojkf7QZ301pzv8nhOBVRkVodRFp8g0Mx6rG3aAqPtmBd91cL+Mr3fk7SYO+qrA17qGUz39kHFB+Tk+63DW386XHc50dPaw/MN0L9i7/Cq+5LPsa+er1TnO+PmRgXANJ9tpjA/EYWe6nKVOSzbyXlqYW9S6DQeeqHUbDlBRCzScXkct09dGpt2hqK7ku341f93RJm7koK/mdLs+5+z2dsTJjRz0q/hSRwTKApvb7Sh1UN2QHKPdPl4h5g6YnfjEskXOliHTud8gFMEFppeJdyjqF3LS4Ww7vUniGbp6zuPV9vfWccLXBP2nYwCK3fs1yZ0apjUcoKPrTKs57W/miXYfD42Bq/lrh7PttaFwYNmz6vbWTLDH2lDYxqNlGbbJVjs/205HfBaTFmgxiPU+tKAD1wNfA54GPpTw/gTwR5IFIiUAAAnGSURBVMH7DwFTva6Zi6CnbEgJLc6o7zE+NYtb7Ks53elLjXTkBc5t5Jng9DLrMO4njE+X426HaMdI2iQRNtLoIBENq4wLdDRPaRZjmJbQJx/mFc74Kl5x44z/HJ91C/4PRXgNp5Jjx+NEBtewUYdiHl4vnOHAmbZwRDt1p5B4u9zCBcn2ABixjEPf/jpO+PmcTi2Pbhtk4j7/CV5upzf6O2G5RgfyMA+thcaz7Rlb1FqL5q0lzGc6hDMc/H6MryaWVzjAdx1UU9x0s+xrDRwZBCtah2F5hbPd+IAQ5vtaDrUXWaN+8uhsJ2z74TUneDnx9StY6uiXcRdK0iAcGjxhWsMF/UQ3VaReVvJKx8AStvFZ9rVm46tuPzcYhJvJgtln2CbD77TaecsoCttG6F6NGh2h8Tjs4vpQgg6sAJ4B3gicDxwB3hL7zL8H9gXPtwJ/1Ou6eVvooWBt575lnTJNOKMdJKzoqLsmbuXcyd3L3A+JU3GSN+Asc12QskmC9HDMeKhdfLtykj85+hvRv+EaQDgDgbN+Hq+2O11fizyxwTUMs4wOrqHFG4rUGk4tE+F43YRpX2adxiJQws4VLfe4GKduGIlZt906ZdwIgLN+fiC4YTjdqkDcJ3i54/tRQYtHMIWD3wq+1/5MaCF3uJ3SBtUB/O5pdRhvk4nRKpG6XjaDi/nU13AqNlifs2CjvxV3j/Y8XiLI8wSnfSKYAYbfT3KfxQer+MASH6zD78TTFc1XWEeX8fX2NeNWeLT8srgvezGsoL8D+LPI/x8GPhz7zJ8B7wierwT+EbBu183bhx5ulIi6DeJukiSRiwphkhWbFPbVraOHFRj36YbfjfrW0qzHtEEiHmoXXj814iPimonnLRqSGL12KMJpm4BSSRCUaDmmdfak/ERdU3DGJ3h5uVUTiE90dhS36mbZ17Ggec5yjG2QSTjLI+peWRaWFtTRj/DUssEmdPnF0xR1BcYt03DwC/+u5JW21Ru1WrPs7ozmM8vxDPE6TDN+4tEwSYNpx8DPOX/7mphAhovsqxMG87BsshwvEc1zWF8XRMI2kxbP0waW+GcTQ1VjaybnR2YG0dlXqDVhn4i667K4L3sxrKC/B9gf+f9m4L7YZx4H3hD5/xlgfcK1ZoElYGlycrLvjMRJ3fwQW9kP/aoOHSIXHZ3jVkF8I0/U+g1H36TFrl6xwPHX4tZjRzxtbJBIEstuVljShoykEMhwAIpam93CI5PI+lthZw87TNRCj0YihJ8JhbmjHGOLh0n1GL6XaQdfyrVSO15KfPoediRai/FNUnvY0RaDn+DRZT7Y8L2VfUQWxc/NibblrNZg2o7d0J+fNUwvvg0+9IFHB9c97PCVfHd59As7/PJgvaDX8RJzc+kBBHGjKD64RweW+DpYdIBImiWH31/NaV/Jdzr64TKDIWUmmXnWlMCwgv7eBEH/3dhnnkgQ9Nd2u25Rpy26e9eY2dT4dXtXzJoabJdi0mp2uFsv/tpNk0eWWVXtKJmEQSIuwMs2U6RETrQjT2JRLuEA2C3Wup8IgF6/FXb26GCW5EMP89HtYKyh9iHESAox7dbx4hE5C2xur0dEFzLDASm++B0Onj/C37XXLgj+RgfWaHhlXx0/w8akbnXYy0DpFbWT5QjrOW5riXHUtx/Z1Jb1eImkUN1eg3t0gTK6wBuvt7iFHrbh6Pdn2XeujhL6YdJCak/3ZQ8a63JJo1s8dq/DdaKi2WFZ5hxL6p59M0PqyYtdYtL7ScMoTxSM/l40ymWWfX3lJ88wsPiZJ8uEJmGgjJfZ7MQn/M0XfdNnJz7RUY5XvfF4x2C+sHaLr+Z0O6oqOjjPsq/9XigW6+n/eNh+9iP0UzZF74IcJN1ZDai0ASde3wtrt7QH4ajARzfD9X1OTM7lN6ygrwS+AVwWWRR9a+wzH4wtis73um6Rgj5IIRYpmkKEdGubeXX+UYpwnlQh3ZU+Mjmgm6Bb6/3umNkNwG8HES8fd/d7zGxXcOGDZrYa+EPgcuBbwFZ3/0a3a27atMmXlpZ6/rYQQohzmNkj7r4p6b2VWS7g7p8DPhd77a7I8+/Q8rULIYQoifPKToAQQoh8kKALIURDkKALIURDkKALIURDyBTlUsgPmx0Hnhvw6+tpxbqPG+OYb+V5PFCes7PB3S9OeqM0QR8GM1tKC9tpMuOYb+V5PFCe80EuFyGEaAgSdCGEaAh1FfSPlp2AkhjHfCvP44HynAO19KELIYRYTl0tdCGEEDEk6EII0RBqJ+hmdr2Zfc3MnjazD5WdnqIws2fN7Ktm9qiZLQWv/YCZHTKzrwd/v7/sdA6DmX3czF40s8cjryXm0Vr8TlDvj5nZFeWlfHBS8vxrZvZ/g7p+NDjdNHzvw0Gev2Zm/6qcVA+HmV1qZotm9qSZPWFmvxy83ti67pLnYus67VzdKj7IcMPqpjyAZ4ndxg/YDXwoeP4hYK7sdA6Zx3cCVwCP98ojcAPwecCAq4GHyk5/jnn+NeA/JXz2LUEbn6B1P4JngBVl52GAPF8CXBE8/z7gqSBvja3rLnkutK7rZqFfBTzt7t9w9+8Cnwa2lJymUbIFuD94fj/wr0tMy9C4+1/SOj8/SloetwB/4C3+FrjIzC4ZTUrzIyXPaWwBPu3ur7j7UeBpWn2gVrj7C+7+leD5SeBJ4PU0uK675DmNXOq6boL+euDvI/8/T/dCqjMO/G8ze8TMZoPXXufuL0CrwQD/orTUFUdaHpte978UuBc+HnGlNS7PZjZF60Y4DzEmdR3LMxRY13UTdEt4ralxlz/p7lcA7wY+aGbvLDtBJdPkut8L/DDwNuAFYE/weqPybGavAf4Y2OHu3+720YTXapnvhDwXWtd1E/TngUsj/78B+GZJaSkUd/9m8PdF4LO0pl//EE49g78vlpfCwkjLY2Pr3t3/wd3PuPtZ4L9zbqrdmDyb2SpawnbA3f8keLnRdZ2U56Lrum6C/jDwJjO7zMzOp3VD6oMlpyl3zOxCM/u+8Dnws8DjtPL6i8HHfhH4n+WksFDS8ngQ+IUgAuJq4KVwul53Yv7hn6dV19DK81YzmzCzy4A3AV8edfqGxcwM+BjwpLv/ZuStxtZ1Wp4Lr+uyV4MHWD2+gdaK8TPAHWWnp6A8vpHWivcR4Ikwn8BrgT8Hvh78/YGy0zpkPj9Fa9r5PVoWyvvT8khrSvp7Qb1/FdhUdvpzzPMfBnl6LOjYl0Q+f0eQ568B7y47/QPm+adouQ8eAx4NHjc0ua675LnQutbWfyGEaAh1c7kIIYRIQYIuhBANQYIuhBANQYIuhBANQYIuhBANQYIuhBANQYIuhBAN4f8DAfZlbOZBOocAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "y_pred = clf.predict(x_val)\n",
    "plt.plot(y_pred, 'ro')\n",
    "plt.plot(y_val, 'bx')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
